Storage Management and Access in WLHC computing Grid

One of the big challenges in Grid computing is storage management and access. Several solutions exist to store data in a persistent way. In this work we describe our contribution within the Worldwide LHC Computing Grid project. Substantial samples of data produced by the High Energy Physics detectors at CERN are shipped for initial processing to specific large computing centers worldwide. Such centers are normally able to provide persistent storage for tens of Petabytes of data mostly on tapes. Special physics applications are used to refine and filter the data after spooling the required files from tape to disk. At smaller geographically dispersed centers, physicists perform the analysis of such data stored on disk-only caches. In this thesis we analyze the application requirements such as uniform storage management, quality of storage, POSIX-like file access, performance, etc. Furthermore, security, policy enforcement, monitoring, and accounting need to be addressed carefully in a Grid environment. We then make a survey of the multitude of storage products deployed in the WLCG infrastructure, both hardware and software. We outline the specific features, functionalities and diverse interfaces offered to users. We focus in particular on StoRM, a storage resource manager that we have designed and developed to provide an answer to specific user request for a fast and efficient Grid interface to available parallel file systems. We propose a model for the Storage Resource Management protocol for uniform storage management and access in the Grid. The black box testing methodology has been applied in order to verify the completeness of the specifications and validate the existent implementations. an extension for storage on the Grid. We finally describe and report on the results obtained

Installing and Configuring Application Software on the LHC Computing Grid

The management of application software is major scientific and practical challenge for designers of large-scale production Grids The Large Hadron Collider Computing Grid is unique in the sense that coupling between application scientists and the resource providers is extremely loose, thus adding even more complexity to the software management problem. After an analysis of the requirements for a Grid software management service from users and site administrators perspective, we give an overview of the solution adopted by the LHC Grid infrastructure to support High Energy Physics experiments, highlighting features and current limitations. Tank&Spark is our server-client solution that extends the LHC Grid application software system and tackles some of its limitations. Tank&Spark can be used as a stand-alone service also in other Grid infrastructures. Here we illustrate the design, deployment and preliminary results obtained

Static and Dynamic Data Models for the Storage Resource Manager v2.2

We present a conceptual model for the Storage Resource Manager, the standard interface adopted for the storage systems of the Worldwide LHC Computing Grid. This model provides a clear and concise definition of the structural and behavioral concepts underlying the interface specification and is meant to support service and application development. Different languages (natural language, UML diagrams, and simple set-theoretic and logical notation) are used to describe different aspects of the model

Data management in WLCG and EGEE

This work is a contribution to a book on Scientific Data Management by CRC Press/Taylor and Francis Books. Data Management and Storage Access experience in WLCG is described together with the major use cases. Furthermore, some considerations about the EGEE requirements are also reported

The Impact of Grid on Health Care Digital Repositories

Grid computing has attracted worldwide attention in a variety of applications like Health Care. In this paper we identified the Grid services that could facilitate the integration and interoperation of Health Care data and frameworks world-wide. While many of the current Health Care Grid projects address issues such as data location and description on the Grid and the security aspects, the problems connected to data storage, integrity, preservation and distribution have been neglected. We describe the currently available Grid storage services and protocols that can come in handy when dealing with those problems. We further describe a Grid infrastructure to build a cooperative Health Care environment based on currently available Grid services and a service able to validate it

Data Grid tutorials with hands-on experience

Grid technologies are more and more used in scientific as well as in industrial environments but often documentation and the correct usage are either not sufficient or not too well understood. Comprehensive training with hands-on experience helps people first to understand the technology and second to use it in a correct and efficient way. We have organised and run several training sessions in different locations all over the world and provide our experience. The major factors of success are a solid base of theoretical lectures and, more dominantly, a facility that allows for practical Grid exercises during and possibly after tutorial sessions

Grid Data Management in Action: Experience in Running and Supporting Data Management Services in the EU DataGrid Project

In the first phase of the EU DataGrid (EDG) project, a Data Management System has been implemented and provided for deployment. The components of the current EDG Testbed are: a prototype of a Replica Manager Service built around the basic services provided by Globus, a centralised Replica Catalogue to store information about physical locations of files, and the Grid Data Mirroring Package (GDMP) that is widely used in various HEP collaborations in Europe and the US for data mirroring. During this year these services have been refined and made more robust so that they are fit to be used in a pre-production environment. Application users have been using this first release of the Data Management Services for more than a year. In the paper we present the components and their interaction, our implementation and experience as well as the feedback received from our user communities. We have resolved not only issues regarding integration with other EDG service components but also many of the interoperability issues with components of our partner projects in Europe and the U.S. The paper concludes with the basic lessons learned during this operation. These conclusions provide the motivation for the architecture of the next generation of Data Management Services that will be deployed in EDG during 2003.Comment: Talk from the 2003 Computing in High Energy and Nuclear Physics (CHEP03), La Jolla, Ca, USA, March 2003, 9 pages, LaTeX, PSN: TUAT007 all figures are in the directory "figures

DataTAG Contributing to LCG-0 Pilot Startup

The DataTAG project has contributed to the creation of the middleware distribution constituting the base of the LCG-0 pilot. This distribution has demonstrated the possibility of building an EDG release based on iVDGL/VDT, integrating the GLUE schema and early components of the EDG middleware

Storage Resource Manager version 2.2: design, implementation, and testing experience

Storage Services are crucial components of the Worldwide LHC Computing Grid Infrastructure spanning more than 200 sites and serving computing and storage resources to the High Energy Physics LHC communities. Up to tens of Petabytes of data are collected every year by the four LHC experiments at CERN. To process these large data volumes it is important to establish a protocol and a very efficient interface to the various storage solutions adopted by the WLCG sites. In this work we report on the experience acquired during the definition of the Storage Resource Manager v2.2 protocol. In particular, we focus on the study performed to enhance the interface and make it suitable for use by the WLCG communities. At the moment 5 different storage solutions implement the SRM v2.2 interface: BeStMan (LBNL), CASTOR (CERN and RAL), dCache (DESY and FNAL), DPM (CERN), and StoRM (INFN and ICTP). After a detailed inside review of the protocol, various test suites have been written identifying the most effective set of tests: the S2 test suite from CERN and the SRM-Tester test suite from LBNL. Such test suites have helped verifying the consistency and coherence of the proposed protocol and validating existing implementations. We conclude our work describing the results achieved